Evaluating the effectiveness of learning interventions: an information systems case study

Currently, there is no standard instrument for evaluating learning effectiveness. While final examinations and end-of-semester course evaluation surveys can be used to do this, they are not designed for this purpose, and there are inherent problems using them in this way. This paper describes a survey instrument, called the Learning Effectiveness Survey, which can be used to evaluate and improve the effectiveness of learning interventions. Learning effectiveness is evaluated in the context of the learning goals of the course (short term learning), and in the context of the overall educational programme and future working life (long term learning). The instrument also provides feedback on the intervention and how it could be improved. A case study is described in which the instrument is used to evaluate the use of peer reviews as a learning activity in a requirements analysis course. The instrument was found to have relatively high validity, but reliability was below acceptable levels. Some interesting results were also found on the determinants of learning. In particular, attitude was found to have no effect on short term learning, but was found to be the primary determinant of long term learning.